Overcoming Catastrophic Forgetting via Direction-Constrained Optimization

This paper studies a new design of the optimization algorithm for training deep learning models with fixed architecture classification network in continual framework. The data is non-stationary and non-stationarity imposed by sequence distinct tasks. We first analyze model trained on only one task isolation identify region parameter space, where performance close to recovered optimum. provide empirical evidence that this resembles cone expands along convergence direction. study principal directions trajectory optimizer after show traveling few top can quickly bring parameters outside but not case remaining directions. argue catastrophic forgetting setting be alleviated when are constrained stay within intersection plausible cones individual tasks were so far encountered during training. Based observation we present our direction-constrained (DCO) method, each introduce linear autoencoder approximate its corresponding forbidden They then incorporated into loss function form regularization term purpose coming without forgetting. Furthermore, order control memory growth as number increases, propose memory-efficient version called compressed DCO (DCO-COMP) allocates size storing all autoencoders. empirically demonstrate performs favorably compared other state-of-art regularization-based methods. codes publicly available at https://github.com/yunfei-teng/DCO .